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The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methodologies for 
use in education, This document is one, of a series of papers and 
regprts produced by program staff, visiting tfcholaisV adjunct ; 
scholars, and project collaborators^all memoirs of a cooperative 
network of colleagues working on the development of new * * 

methodologies. ' 

■■■■ t ' . \ ■ ■ 

What procedures are available for making educational decision, * 
more systematic? Wiis primer contains a collection of Q 

(a) prescriptive guides for clarifying problems and objectives, 

(b) suggestions for identifying possible solutions, (c) criteria 
for judging solutions, (d) methods for measuring values, and 

(a) strategies for combining these elements to reach a decision, 
ihe thoughtful use of the procedures outlined/in this primer can 
formalize and improve decision making within educational 
organizations. ~ . " 

■ Nick L. Smith, Editor '* . 

V«& * Paper and Report Series; 
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CfiAMfcl: PRELIMINARIES 

=v V'- 1* k%#p six honest serving men 

/■ s 5l'm?hmY taught ine all I knew} ! 

;. " V ■Jfcaeir ftarteitfe What and Why and Whan 

:\ ' ■ \ ' ' : tod few andHhere and Who, 

t ; ' Rudyard Kipling 

TOM? is This -BgSjfl^gJjMbattt? J 

DAP* DAP r^^ptesenta DscLsion Analyses Procedures, or, if J^U 
prefer a le^s p^e^aic -explanation, dap is short f or "dapper", ^ 
which mean** n%afgfc mn<n styliflbt The procedures presented in thi$^ 
primer are not t=>rigiaal. Ifey are: taken from many disciplines, 
such as opetaticrDns teseargh, psychology, economics; and • „ 
roanag ement sci«cacef an4 present a more, systematic approach to 
decision making than customarily practiced by most educators. 
DAP are prescript tjv^ and concerned witji how tio make decisions 
rather than b%ieig descriptive and concerned with how decisions 
ate usually m%d^* O&P inslude guides for clarifying problems and 
objectives, sug^gestiQns "for identifying possible solutions, 
criteria fot juc^ging solutions, methods for measuring values, and 
steps for putt in=ng it all together and arriving at a decision* 

why ^his '■Btftyya , wait gap? 

' Solving ptofcliflte " aftdirting decisions Is dif fjg grt^ and 
stressful wo^H- AS Idnusi the Peanuts ca rtoon character said *o 

Charlie^Browns 4 

* • ,=..,.. * : : / -•• 

I don't like to face problems head on. 

I think tfcebest way to solve problems w 

is to avoid thenw v.- . :■ ' *■ 

(Tfcnis is a distinct philosophy of mine. 
* - > . Nc^> problem Is so big or so complicated 
. _ that it can't be run away ^rom, 

** • * j \ , . , * ■ 

People Are a£C#«^ted by #at we decide and do, ■■. Problems are 

complex. Vet* fiLn our professional, role's, judgment and decisions 
must be iii^ tflrf not **yiuway f rora* . Hany of our actions ate - 




"Hot^td^ this tfkn* ^(/l^otigfiesf dtc&oru 



Unljjd i jJie i^a§£4|!g. Friendly' Slclgft, 0ete^fe©£ 



mora instinct iva than r^agoo^d* But our aonmi^c^ genia should fceil 
us that We cannot always tat^ on que towwm ptasti ' ffe bWlipv^ > 
that the* dap diicribai tn tfaie • primer «feah imMtov« decision 
making, Althciijh some of tfa^ procedurea iti^ol^^e afithmetlaai 
operations. thatyieW riUiflefLeal estimates of : tfcae value of 
decision *alfcfttnativ<Mr fch% ^Wf ail aim of fch§ ^AP ii npt Aumb^ri . 
but grantee I jiiight into fchte problimi we £ w% and the possible 
solution? we lip to' Solve fcrJiera, : . 

ShQiild a§ptf m X3&md^ ' -: m ' - , . " ■ ' 

V ^ r ^ T ^^^^^ ^ - • ■ -;. y _ 

W© b^Xiev^that at laa^fa. soma of the pfcgQaaucai are 
appropriate feriiirtcijig any Ljsiiportant. decision* Although per ^cz» na i 

decisions ( foi timinple, whig-fci progfam to #nrol£^ into? how to ^ 
distribute youi tirae among o^ompating aatiyiti#M} are eovaffed toy 
the procedures, the Qf iifititiQn is toward organizational concerns 
(how to fun tti& language lab-moratory, which tejct^iDOok to pufchas . 



*^he procedures are, )>«**e3t suited when people wish to reason 
tjoge therein good f ai ~ "th as they identify, and evaluate goals and 
decision, alternative =ss. The parties can have different coles and 
values/ the goals ea^m be ill-specified, the information, can be 
inaompletp and iroper^^eoi, and the criteria by which the adequacy 

of any solution is t*=o be judged can be many and varied, 

j ■ ■ . ■ " ™ • *- * " 

WHO is this primer f«saor? • 

Decision makers. Although all of the examples will be 
educational/ anyone , ^^ho participates^ in making decisions may 
benefit from the prlxrmer and the DAP - " . '• " 

VHE5E Doob Coat Analogy sis Fit into the DAP? 

Decision makers, ^should be clear about their problems, 
identify attractive er=3eeision alternatives, and evaluate the 
expected donaequenai^s of implementing these alternatives* in 
evaluating these pdsr=3ible courses of action, all important 
consequencei should E)e considered. Financial cost is usually one 
such consideration* student learning and satisfaction, teacher 
acceptance, space a ncrdl other requirements, and the ^raatch to school 
ph ilosophy/6b j ec t iye^s are among many , other consequences that can 
serve as criteria fiO-^^ evaluating decision alternatives s Thus, 
„ the DAP arg broader *^^han cost analysis, both because they 
enconzpagi a greater Export ion of the decision making process, and 
because they treat east as only one of many criteria to be 
applied in the evalu^Mtion of alternative courses of action* ( 

HOW Might This prime^^ be used ? " ? /' 

It is suggested ^shat the reader proceed through the entire 
.primer. Checklists* steps to be followed, examples* and other 
material that are most suited as a future reference are placed as 
separate figures for easy identification as , needed. 



* ' * * CHAPTER 2s ABOUT DECISIONS *** 

.« -. ■ ■ ■ . r if* 

Decision Con texts ; - 

Decision^ and the decision process have political, morale . 
economic and soc io-psycholog ical components. 

Decisions sometimes take. place in highly political contents 
in which choices are based oh influence , power, position, 
ambition , pressure , badger ing > negotiation, and gamesmanship * 
Procedures for decision* making in such contexts are not addressed 
in this primer* Conflict over goals and values can occur in the 
situations considered here* but, such conflict is expressed in 
settings more akin to a participatory/ democracy in which 
consensus is sought even if it is not attained* s 

Decisions take place in a moral context in which choices are 
based on Oughts and shoulds, duties and obligations, and rights 
and wrorfgs. Moral and ethical concerns manifest themselves in 
the DM* as * they influence " the choices considered and the 
s evaiuatioh=jof these choices* '■.=-.• 

Decisions take place in an economic content in which choices 
are based on benefits and. costs* supply and demand, income and 
expenditure. As stated above* economic (costX implications are 
treated in the DAB 33, one of possibly many consequences of a 
decision to pe evaluajbed, -•■ . 

Decisions take place in a soc io-psycho logical content jtn 
which choices are based oh the interplay among, actors— on their 
values and needs and feelings* Such interplay figures heavily in 
the DAP described in this 4 primer* , ; . , 

Decision Making in practice ^ 

Two extreme decision making styles can be described, In one, 
the decision techniques are idiosyncratic , ad hoc , and 
inelegant. Decision makers rely on intuition* good judgment, and 
experience* In the other/ the techniques ace mathematical, ^ 
mechanical, and elegant. Decision makers rely on analytical 
thinking, mathematical models, and classical rationality* 



As one writer cynically puts it, in the first approach decision 
makers act before thinking, if they ever thinks in the second 
approach they think before acting, if they ever act, = 
Psychologists an^^so^iblogists are fond of describing how * ^ 
decisions emerge, and they conclude that* although most " # 
procedures followed lie between the casual, flying-by-the-seat-of- 
the^pants and the studied, number-crunching styles characterised 
above , more often than not they are a laser to the farmer than the' 
latter. >. ' ■ ' * 

The present can be the enemy of the future , tod the way 
important dec isions are now made need not be the way they should 
be made. The DM shift the emph*asis toward the analytical 
; style- But the power of bote styles is recognized, and DAP 
incorporate the seasoned,, experienced judgment of the first with 
the analytic, rational style of the second . The DAP back away/ 
however,* from the casual, capricious approach of some decision 
makers and from the preoccupation of operations-research and 
management-science everts with sophisticated mathematical models 
and with estimating probabilities. 

DW consider decision making as a problem solving task in 
which problem clarification, information, and creative solutions 



are required, ^and,-in which consequences and utilities merge to 
produce an evaluation pf the decision choices* DAP, attempt to 
'" ..clear "away the verbal' underbrush and expose the path to * 
decision," (Bross,, 1953, p. -257) They force concentration on 
the essentials of the decisions options, criteria, and; values,. 

Implicit in the PAP is the belief* that the making 1 of / - . 
important educational decisions' should be ; a cooperative, effort. 
"Every competent boss has a staff whose, views he respects" arid, 
whose values he regards as relevant." ^Edwards, 1977 / p, J27,) 

■ / * * --'..= . - * . * 

Whether or not the .process is cooperative ,* it does, involve many 1 

parties, . . t - . 1 • r ". j 

; . .'decisions that become official when ratified by the 
managers assembled in one room are the result of a" 
collective process that has taken vplace in many rooms.; 
And, after the "manager"' has spoken, many voices infinany 
other rooms will, determine what happens. (Gronbach7 
..; . 1980/ p. 85.) /..■' [ . >, 

Having others participate in decision making has pluses and 
minuses, "The rationalist, ideal of efficiency is in tension with 
the ideal of democratic participation," (Cronbach , 1980V P- 95) 
Duke fet al. (1980) studied the perceived costs and benefits of . 
teachers' involvement in shared decision making. Teachers saw * 
the major cost to be increased demand on the^r time. Better 
feelings of doing something important, a strong sense of shared 
ownerships and greater satisfaction' in fulfilling their "right; to 
participate" were listed as benefits. " . * 

Radford (1977) suggests that "decision makers dealing* with 
complex models -adapt procedures that include: 

(a) simplifying the problem and selecting ,a course of • 
action that is judged good enough rather than optimal; 

(bh avoiding uncertainty 'or taking steps to reduce the 
^effect of uncertainty on the outcome of* their decision; 

* (c) concentrating on incremental measures rather than 
those involving fundamental and large changes i and, 

(d) placing emphasis on communication aind participation 
„ in an endeavor to reduce the effects* of conflict.-, . (p. 16) 

• * . 

* The DAP recognize the usefulness of practical/ procedures like 
these; But they also affirm the wor£h* of, more analytic 
procedures in which personal values are macie explicit and the 
merit of each decision, alternative is judged systematically. 



Steps in the Decision ProceBg"™" 1 ^ 

• * ' Descriptions of how fc© i^ke^dWisiOTls s how to .solye 
problems; or Row to think** evaluate # ~p lan or engage "in, systems 
analysis aU seem very much ,V alike "7 Fo^^Joffirt ^Deway the questions 
to ask ares What*S the problem? What are the. alternatives? 
^Whiell ai ternatlye is best? For Herbert S imon # the afea#esf#fce ; : ^ * 
intelligence gathering* desi^ and analyzing v • 

: al*er natives } ; >: aftfl choosing . The; DAP assume that the; decision ^ 
steps are similar to those shown in Figure 2.1, - 




Figure 2.1 ; Example of a decision pr^c 




Clarifying the problem at hand can be the most important ; 
stepr because a solution to the wrong problem, no matter how. 
creative or elegant, is worthless. According td Dewey, "A - 
problem well stated is half solved." The decision problem will 
more likely be well stated if it is written out. 
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- -v.. A tension exists between stating a decision too broadly f so 
* that it o4n»t be dealt wifch\ effectively, and too narrowly, so 
r V that attraotive?c©urse3 of action are excluded .f rom consideration^ f 
The umbrella decision, how to improve the* school, would be more a 
manageable if it were* limited to improving some aspect of the 1 
school, such as teacher mora lb* tin the other hand, - a decision 
about which school to close may be too narrow and restricting. 
* A more appropr iate quest ion might be how to cut a spec if ied ; 
dollar amount from "the schooirBudget. ••' * *• ? v' > . ' s 
Asking „ "Why? w is a pow#rf^l heuristic f pr__uncove r ing 
7, underlying issues and values and for exposing those decision . • 
problems toward which one 1 s focus and effort should be directed, 
i^^Wyfj do we need a change? - Why is there ~a -problem?! r Why do * we Iwant : 
to make this decision? .Whea "Why?" is applied, to a question such 
as which computer to buy, the purposes for the purchase jare 
.elicited and a more basic and less restrictive question might be 
revealed, such as, v"How should the block grant allocation be 
spent?" -\: '" V; : ' . >; - ' : " : " : '7 V • Vr . - ' ' : r - : " ' 

Frequently questions are phrased in terms of goals and 
objectives, ; suqh as, "How can we increase student learning?" 
Decisions deserving priority are those affecting the schools % 
most important goals* To avoid closing off options needlessly/ a 
statement of goals should avoid mentioning possible solutions* 
Thus, it would be; better to ask how : to -keep the classrooms 
comfortable rather than how to c^ise the temperature to 68 
"decrees, because eliminating, drafts and increasing humidity might 
be more effective solutions than raising the temperature, 
" In short, the way in which the^ decision problem isf stated 

limits the solutions or alternatives that are considered - and , ' . 
, consequently, -the way in which the decision is resolved^. But 
&ven a "well-clarified decision problem admits many choices of 
_ action* ' Since a "decision" itself is a choice among alter- 
natives, it can be no better than the alternatives considered* . 
Suggestions for identifying alternatives, the next step shown in 
Figure 2.1, are offered in Chapter 3, - v 
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% : ; ^aluaiion or ^€ altaenativas encompasseff two ;-;stepk • £ 

firsts dealt with in Chapter 4, is to determine the factors or 
criteria N to server as . the basis for the evaluation^ The secpnd is 
to apply procedure s for a r ri v i rig ft t a nume r i ca^ indicator of ^ - 
value for each decision alternative. _J?plCy ^this„#tpp#r dozens of \ 
decision; aids . have been Suggested. Models and techniques, , 




theraatical and otherwise, Abound. (tee Hwang and ¥oon# : 1981 # / -; ; 
taxonomy of Multi-Attribute Decision Making ^(MADM) .< 
techiijques, many of which use Utility Theory (MAUT) - A ffet of 
procedures that is analytic, defensible, and manageable is ;/ 
^described in Chapter 5- , . • • • - : vV, 

*.:/^^ 

is an additional step in the decision process. (See Figure 2.1.) 
Discussion of how-to carry out decisions once they are made, 
however, is beyond the scope; of ,this primer. • • 

The feedback arrow looping back to the first step is included 
to remind: us that the decision-making process i s tfcDt terminal y 
the' end. point of one cycle can be the starting point for 
another. The work of educators proceeds incrementally. One _ 
decision begets another . ' - 
Also presented * in Figure 2.1 -is the view that information, can 
apprise all stages of the decision process. The terra "informa- 
tion* 1 is used broadly to' include not only facts-»-nuraerical or 



otherwise, current or historical— but also expert opinions, value 
preferences,- and prediction of outcomes* Information may be 
gathered by any of a large number of techniques— formal or 
informal, objective or subjective- An excellent resource 
guidebook for educators on information useful /for decision making 
t has recently been published .{California State, Department of '.. 
Education, 1983.) ' \. :. 

DAP seek to pool the information available* and to apply a 
.wide range of data, experience, and judgment to. the decision 
process, One objection to the DAP being presented in this primer 
is that the input facts and opinions may not be very accurate. 
"Garbage in, garbage out" is the caveat frequently heard . when , 
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analytical models are , d iscussed • -However/ these fallible input" 
idata are the same ones that woiilTd be employed even - iess^ Z- — 

analy tia^method - were^ trsed y ^ . ; fc . • - ; ^ \ V 

Most decisions are not of sufficient importance to be made- 

• following ^thp steps shown ih Figure' 2* 1, Making "dec is ions ; 

; "anai^tieally costs money and time • The se - inve s traeh t s must be- : 
balanced not only by any incremental gain that might accrue from 
identifying and choosing a good option over an inferior one r but 
also by any side benefits that are expected to result from \ V LS; 
following the DAP. These benefits include the comfort that V = 
corroboration of unaided intuition can provide |/ the increased * 

. insight of .the staff into a complex.; situation and the trade-off s 
involved;, and the improvements in communication with and 
acceptance by staff of a particular decision when the are^s of 
d is agreement and the justification for the chosen option are made 

• explicit . ' ^ . .'. v • . • ' ' p ; ; . . - 

Of the six decision-raakirig activities shown in the rectangles 
of Figure 2.1, only the - middle three, ar£ • d iscussed in further 
deta il in the chapters tha % follow Th is, se lection is not 
because these activities are more important than the others, but 
because a more limited focus . of this primer is desired. 
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What is a double petunia? | 
A petunia is a f lQwac J-ike a begonia* f 
A laegonla is a meat 14- ke a sausage _ w -"">;- • 
A;;Sf^age^an4*battei?y is a or line • r 
Monkeys crime trees . ; • : , * .- j. ; ' , .'. ,^2^:-.. ; . : [ m : ; 
Tre% 'i a crowd. , 4 - " • " 

A GCpw i crowd in the morning and made a noise- V V 
A nbise is Ion your face between your eyes. *\ 
Eyes is oppps x ite' from nays. ^ ..^? 

A oplt: nays. "■>'■ '■ ■ '•' •* -' ■ ■ \ -•" 

You go to bed with a colt , and . wake up in the morn 
with ;a case of double petunia. -. \ 
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A decision implies a choice among alternatives . No decision 
is 7 necessary when a- single option is ava ilable * ( "Alter natives " , 
"options" , "choices" V " strategies", "potential actions", and v 
"solutions" are use4 interchangeably. ) The alternatives do not 
have to be distinct educational programs. ^Having no program at 
all is an alternative. Different levels of financial support- of 
the same program are # alternatives. - , J 

The identification of i bnly infetior x alternatives will, in . 
turn, produce an inferior decision, as will failure to consider 
an pu tstand ing strategy. Because the choices considered limit 
the quality of a decision, it is important for decision makers to 
know techniques for generating and identifying such choices * 



Suggestions for Identifying pec ision Alternatives 

" *, Making educational decisions, differs from selecting greeting 
cards in a stationery store, because with educational decisions 
all the options are not conveniently laid out, Technically, an 
infinite number of options exist, and only a small fraction are 
known* to the decision maker. Eight groups of ideas for 
generating and identifying alternatives are discussed below p'-^T~ 
listed in Figure 3.1 ■ 
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: '1* jJh^ 

2, Consider choices nfade in sxmilac or analogous 

*■ •.' ■ "' /.■«*" ...'■ = ^-' - j - r; -v : ,. _. "• . • ■■ ,.- .." ' .•_ -1 

; : V " /.':"» > ' ^i" : ;^';vV; ;V i " '".•. '= *■".'."."*-. '-. - ' '" v; • -. '• \'-« 

3, - Employ a checkXist of quest ioria* ? /m/ J 
■y-Am ; ^allen|f# alleged v :c6ty&. tr a int a . • ^ 

- Avoid two-option scenarios. -Vv- t ' » : 

6. Use your imagination- ;V; 

7, Solicit advice from others. (See Figure- 3.3 and 
• related text foe one approach.) " L - ^ - 

v V :■ r .'•:;<"' • -:H : 

\j 8. Piggyback on the ideas of others. /' 

f Figure 3,1 Suggestions for generating decision 
alternatives. - 



: • •• - : • 1. Understand and analyze the decision to be made ; Options 
flow from the decision problem and. from the goals lurking behind 
it. The decision and these goals serve-as^ guidepostM and 
boundary markers for locating and assessing decision choices 
the strategies for attaining the goals^ Dissect the goals into 
subgoals and think of ways to" achieve each subgoal. Analyze the 
.situation. Ksk such questions as. What are the parts to the 
problem? What must happen first? ,Next? What is required? How 
'can it be attained or made to occur? What depends on what? , Why 
\is there a problem? Appreciating the reasons for .the decision 
and being able to sense what in Jtyie present situation is 



unsatisfactory, or in heed of change can suggest solutions for 
remedying the problem, * ; 
• =- •■*" 2. Consider choices made in similar or analogous decisions, 
^re there historical precedences? Have you or others made 
similar decisions successfully before? (Don't hesitate to call 
on your own past experience. ) What general class of decision 
problem are you facing? (Look for the powerful verbs in the 
problem statement,) Is it an allocation problem? a search 



- _ , %/f-: ; ^ v. ; , \. : ^ 

eoniattunication problem^ Gather infonoation oii^hqw' lueh; problems 

are dealt with by othks^vei^% ^ <>£ : 

• v- "v^ / : . ^ - . ^ * ^ - w V -V ^ • '.. " ' n \ ^ "'• "•' 

^education. *• ■ • v .\ % \ \^ v : : - J - - .- \> v * . "* : 'f ■ ■ *' 

\ - - v ^ ' - ^< - * ^ • J w - ^ - : 

• v . ; ° 3^ to ploy a checfrlijsfc of questions. A C systemat ic approach' to 

; S(eTOrati^g : options 



^quaiitlqh^^^ existing or previously thought of^ 

: a fcr iteg ie s - Some questions ; we re • ia^ntif - Other v 
!%^ue s t ions . r de s ig ned .to suggest var iat ions* on pr 0 pent practice or 

product s> are shown in Figur^ * 3>2, . V?^§^ftS^^' ; it- , ; ' ." 
; I--'-. 4v Challenge alleged eonstra restrict v; 

^tiohs suppress you from generating attractive solu Are the 

^ constrain^ ab^ut th 

real? For example r is' the decision really; about; which^ ^teacher to: 

{lire or about sta££ing-pitter 

dollar amount the true upper limit of the re source s # or can - other 
- funds be re ioca ted or secured through avenues not .qpnsii^eteed? " ■> 
Are the policies> " social mores, and ^biic^^iniw^ttat t v : 1.^-. 
apparently ruXe > out ; selected processes imagined or real? , Restate 
constraints in terms of functions ratK^r ttanrin terms of : 
objects. > ^ Enclosure* is less constraining in the number- of 
admissible dptions than is "fence". How to maintain' "discipline 1 * 
♦is a less constraining goal than reducing "noise** . 

_ 5 . Avoid . tVo-option scenarios. Thinking in terms of f 
dichotomies is commonplaces yes or v no; buy new uniforms or not 
buy new uniforms; hold a junior dance or not hold one tRis year? 
have a 4-day week or a S-day week; adopt the new program or Jiot- 
But rarely do only two options exist.* Many attractive choices 
are usually available, and they should not be overlooked. For 
example , in addition to adopting the new program or. not adopting 
it # other choices are: (a) postpone the decision; (b) adopt the 
% program on a trial basis (limited time, limited number of sites, 
or limited in degree) ; (c) adopt the program contingent on othet 
conditions being met; (d) adopt the program, but not until the 
year after next; and (e) adopt the program in principle f but with 
. some features changed* Classes of options beside "yes 11 and "no" 



Put to • ofcher'^ uses? Mew ways to use' aar i#7 0£hte£ iimmm ^f- 
modified?, A . - ' VV- ■ . ," '• • »- : 



Adapt? What else is like tfeis? What other ideas does this 
suggest? Does past offer parallel? ? Whdt could I copy? ~ Whom 
could 1 emulate ? ■•" *' ; < : v ' - 'rJ ^ • 



Modify? Nefetwis*:? Change meariirig, fcblor # motion/, sound, odor., 
form, shape? other changes?* ; ■ 



Magnify? What to add? More time? Greater frequency? ~ 
Stronger? ' Higher? : Longer ? X Thicker? - Extra value? : Plus - 7 I 
ingredient? Duplicate? Multiply? Exaggerate? ^ - ^ 

Minify? What to subtract? Smaller? Condensed? Miniature? ^ • 
Lower? Shorter? Lighter? omit? Streamline? ^Split up? 
Understate? . _ -J....- ■; ...... 1 \ 

Substitute? Who else instead? What else instead? Other 
ingredient? Other material? Other process? Other power? Other 
place? Other approach? Other tone of voice? — . - 

Rearrange? Interchange components? Other pattern? Other . ' • . 
layout? "Other sequence? Transpose cause an$ effect?- Change 
j>lace? . Change schedule? ' 

Reverse? Transpose poslti ye and negative? How about oppos ites ? 
Turn it backward? TUrn it upside' down? Reverse roles? Change 
sHoe£?~ Turn ' tables? ^rh other cheek? ^ - 

Combine? How about a blend, an alloy, ah assortment, an 
ensemble? Combine units? Combine purposes? Combine appeals? 
Combine ideas? 



Figure 3.2 Questions to elicit decision alternatives* (From 
V Osborn, 1963, pp* "286^287.) 
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incliid^ doing nothing ^ 'applying a ^dif f erent iemecly/ applyi^ ihany^ 
, remedies, starting over, and getting rid of a troubling sittfetien 
e ^Pee your £matii nation* -Be original. Be /far out* consider 
'ideals, DayTdream, Ask, ^rhat if 3 Contemplate; ? speo^lat©,r * 
v theor ize ^ponder - Br eak < £ r ee from the traditional. Let th£ v 
s^stzfeam^ofc-'TOMfeictoeness- flew, v^>creatiy<i # - : " i ^ " • ' ; 

, 7* Solicit^ advidb from others. Ask other people for ideas * 
Experts are good sources; but dim" It limit yourself tp them. Other 
admiftistrators f ^eaehe^i, students , special interest committee 
v members, and <| is ittter es ted, par ties may all be helpful* : Brajn- . 
storming* pnfeV ap^rbafch for; drawing out suggestions from others, 
is discussed In the next pairt of -this chapter. 

building oh* jhe? "choices suggested by others are associations, 
var i at ions , r and^ combi nations S- j Free associate and/or pfr^dUce* 
chain reactions, (Recall the answer to the double petunia 
question^ ajty the ^^inning of the chapter.) Exaggerate. ^ • ; i r . 
Figure 3.2 lists questions that encourage variations* - .r^y^- ■■, , 
. Combinations can emerge toy pitting together parts of i;Wp (or more 
> strategies and forming different arrangements • For example, an 
instructional program contains materials, staff, students, / 
content, etc . List elements under each component; that is, list 
the options for materials, staffing, etc, The procedure then 
consists of generating new suggestions for an instructional 
program by forming different combinations of the elements, a 
procedure which is like dressing up a paper dp 11 with various , 
combinations dt hats, dresses, and shoes. / ^ i - . 

Bra ins terming = ' 1 . * 

If two heads are better than one t are several heads better 
than two? Many would ^nswer "yes 11 , at least if the task is the 
creative one of identifying decision choices . Brainstorming is a 
procedure in which several individuals, thinking together, offer 
as many choices of action's as they can, / w * 

Brainstorming is recommended when many solutions ofc 
, strategies can be discovered or invented, when the preferred 



*6ption^sTibt purely a raitCer of audgmeht* or when^the - : V- 
alter natives are ndfc already, known i * Thu s , brainstorming ..; is 
appropriate for a quest ion suc v h as : 9 What is the best way to, \ 
Brains terming is not appropriate for questions sueh as>/ When is a: 
the best time to* or. How much should ^w#i ihe decision . 

v problem should be sufficiently narrow ,that solutionsj^afi; be ^ 
forthcoming , r Howe ver , if the decision problem tends to be broad 
in scope, it may be possible to divide the problem into component 
parts,- with a brainstorming effort directed at each pmf t in turn* 

1 1*"- is recommended that the brainstorming group consist of &V 
leader, a manageable number of panel members, and a secretary or 
tWc^to record suggestions, ks (It is not unusual for suggestions to 



be/bf f fered.at t oo..f kst a pace jEor one .secretary to keep up.) 
Suggestions aife[ apt to be richer if the - panel members possess 

' different - backgrounds and interests. Some experts on .'.- ;■* 

brainstorming caution against including in the session the chief 
administrator or some other person holding a superior ^^at us oyer 
the others* The fear is that the presence of such a person will 
inhibit the other paifticipants, / making them less likely to > 
suggest alternatives that are unconventional, playful, or 
otherwise likely to be viewed ^ritically by the superior* 

In preparation for the brainstorming session, the leader ..- 
should inform the participants of the decision problem and the 
task, for to^ to identify a large number of , :. " 

decision options) , provide a few (but not the best) examples of 
solutions, 1 and indicate thq procedural rules* These rules ^ 
include prohibiting any critical ccmunents, which. tend to disrupt 
the creative tempo of the" group . Questions are also frowned 
upon, because they frequently are evaluative in content* Also, 
ideas should be stated concisely, because evaluative judgments 

j are apt to creep in during the elaboration of one 1 s suggestion* 
During the, session, toe leader's goal is to keep ideas 
flowing. Free-wheeling suggestions -and shooting wild are 
encouraged, for , as one writer. put it, it is easier to "tame - 
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down" than to "think up". Those who promote brainstorming take 
the view that quality of ideas will borne with quanti^ # and that 
types of alternatives, silly or serious, far-out or feasibly, 
should be voiced..' One tactic designed to increase the number of 
options offered is first to solicit all 'ideas that piggyback on • 
the our rant suggestion before entertaining new ideas* To foster 
production, the leader should be prepared to deal with lulli by - 
such ploys as repeating a few randomly qhosen, previously offered 
suggestions, or by mentioning verbs that critically define . the 
problem to serve as trigger words or seeds to plant hew ideas. 

, An overview of operating principles for brainstorming is 
provided in Figure 3.3. % 7 " >' 

Decision Selected for Brainstorming 

a) Yields many solutions not readily known 

b) Sufficiently focused 

Composition of the Brainstorming Group ""' 

a) Is diverse in background 4 •' \". • 

b) Is homogeneous in status/power s , . 

c) Has a manageable number (for example, 6 -420) 

Chairperson's Actions \ -.■ \ 

: a) Prepares panel before brainstorming session ^ 

b) Discourages, Judgments and questions 

c) Encourages ideas and persistence 7. 

d) Sees that all Ideas ate recorded ..." i 

e) Solicits after thoughts / ' : :: 

Panel member's actions * ' ' V 

a) Withholds judgments and questions 

b) States suggestions succinctly 

c) Generates ideas freely 

d) Persists at the task 

.■■ ■ '..•*'■■ - ■ . . . ,_.= .- r ' 

Figure 3.3 " Overview of brainstorming operating . . ' 
v principles. : : 





— Panel members repbr t that participating irt a br a ins t0£mi ng V 

/ session is both enjoyable and hart work, and can hav^ such side 
behef its as incr eased morale ahd f legibility in ^thinking. 

' Brainstorming can be useful not .only to identify decision ; . * ; 
; alternatives, but Whenever suggestions are needed--f or exaraple> 



T - : < ^^ta^id^tify Criteria for judging the alternatives or to identify : 
, V ? reasons for taking a particular course of action. , : v ; 

: - Taming the Ideas V .' ■ ; . :-!•". ' 'V " 

Regardless of whether decision alternatives are arrived at . 

through brainstorming sessions or through the other means 
- suggested, these choices need . to be reduced to a manageable x 

number; Wher^ applicable, the status quo and an alternative that 

is only incrementally different should be included. The latter 
.' ; is suggested for consideration, since studies of decision making 

find that many decisions represent strategies that differ only 
'-• ■ ' marginally from current practice. The remaining suggestions can 

be grouped, and the most promising option in each group selected 

for further consideration- How such further consideration might 

occur is described in the next chapter, 
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- r ;.; ■ CHAI^R 4. DETERMINING CRITERIA . • - * 

/" Once decision alternatives have be^n "identified r ?a ipgiaal 
next step in the decision process is to\consider the merits of 
each * * DAP invo Ive eya lua t i ng how , well each' of -. the decision ■ J 
alternatives measures up against important yardsticks or factors 
f of yd lue « in this primer , the factors , such as achievement gain 
and cost t that are used to evaluate decision ■{ choices are called 
•'criteria*' , or "value dimensions". "Objectives'* is not used here 
because the. important criteria may be more, inclusive than how *?; 



well objectives have been met* for "criteria" that 

"have restrictive meanings are "attributes" ^"characteristics" , 
"aspects", "components", "features"', "properties" , and 
"performance parameters 11 - : 

Whether a criterion is consider ed".important depends upon what 
the decision choices are and upon who is judging the potential : 
actions. If the alternatives are competing instructional 
programs, a school administrator may most value cost and 
community acceptance ; a student may give priority to other value 
dimensions, " If the" decision choices are various disciplinary : 
actions, different criteria, such as fairness and congruence with 
exisiting policy, might be viewed as most impor tant - . ' 

Identifying decision alternatives (see the previous chapter) 
differs from determining criteria, in that originality and 
quantity of options are more important when the task is to 
identify potential decision choices* The best action to take may 
be one that is unlikely to surface unless effort is made to 
generate; many original options. When the task is to determine 
criteria, however, only the criteria important enough to affect 
the decision are sought • Only a handful of value dimensions are 
likely to be needed, If a potential criterion doesnVt come 
readily to mind, chances are good that it isn't, important enough 
to be included while implementing the DAP* 



li 



? s Five suggestions for < determining 1 criteria are listed in 
Figure 4*1 and are ' discussed .below* - 




Sugge s t i oris for ^Determining Cr iter la ^ : " ■ 

1 9 Consider- the best and worst features of -each decision -< , f 
alternative « ; The special advantages ^hd disadvantages of ^5 * 
Option (tor example, availahility %and size) pan suggest important 
value dimensions* Consider also the consequences of NOT, * 
following a potential action; ^ '-. [ 

2* Consider" past experiences. Were decisions like this made 
before by you or others? What cr iteria used then dt now seem 
important: in retrospect?; If .the hdec^ product or- 

to implement a program, what , attributes are usually looked at in • 
evaluating such products or programs? 

• .3* Start with broad concerns? then narrower ones One, - 
strategy is to determine broad criteria (for example , 
effectiveness) and then to specify ^component dimensions that most 
con-tribute to the broader cr iteri'a-- - This strategy leads to a 
hierarchy of criteria, with specific criteria emanating from more 
encompassing ones . : " \ * " 

4. Employ a checklist of criteria * Frequently used cr-iteria 
are mentioned in Figure 4 . 2. Reference to these, or other 
criteria suggested by them, can ..help prevent overlookingvan 
important value dimension. Note in Figure 4 . 2 that. DAP treat 
risk or uncertainty as a value, dimension in its own right. . If - 3 - 
the chances of positive consequences from choosing' one decision 
option are more uncertain than irom following for example , a. 
more s t and ard^cou r se of action, this greater risk would count 
against the first option when the twp alternatives are evaluated . 

5., Solicit " advice from others , * Who will be affected by the 
decision? What do they see.as important criteria? ^Obtaining 
value judgments from groups of individuals that represent their: 
true and informed opinions is not easy, .Some cautions and a , 
,dencription of one approach for soliciting and consolidating the 
views of several; individuals is discussed below, , 
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1. Consider the ,best ani Wfar at features of each 
- decision alternative * ^ 



2. Consider past experiences • ' \ ; > > ■ 

3, Start with broad concerns; then- narrower ones • 



': PA 



^4.^^nplri^ a^hecklist o^ criteria* 



5. Solicit advice from others* 



Figure 4* 1 -^Suggestions for determining-decision 
: ' ..' criteria- -' 



21 



27 



ERLC 



1. Character is«/es/Features 

• . -v v ; i » H acceijfebility !/ K 3 ^ . '.I- u\ ,'; ^ ,r , \.;.;\ ;* • 
^ v attraetiy^hess ;' V -': = ; 7""v ; ; v ;- : y ' v- : 
consonance with *existing policy, 

; : .leverage . - 

■r .= ;'. j<- protection of; rights 7 Syj- y. ^ 

• ••' -V. p&pQ^^ 1: x--^\'-" : - •;/ - : 

quality - : 

2i Requirements ! > - • -' -•^- v • : o '}'•'."'. 

. : cash outlay ; ;: _ . v y ■ ; • V' ." ' . y^M- 
, ; facilities/space;/ ,...:....-./._. y ' ;y.-y _ 
' , - mate r i als/4upplies /equ ipmen t/ma in t e nance 
personnel/xraining 

3 . Implementation a 

availability 
ease of implementation 
political viability ■ 
transportability f 

4. Effectiveness (or- changes in) 

* achievement V • y " 

attitude/appreciation 
attr ition/rt^Mivism rata 
behavior 

felt or meit needs ; -. 

interpersonal relationships 
longr-term effects/durability 
numbers and audience served 
.• program content 

: satisf action i - . , •;. : ' ■ . • 

/sensitivity/awareness 
V y side effects/spin-offs 

structure of the organization or program 
value to society 

5- Risk or uncertainty with regard to 

! characteristics 
• ' requirements < s ;.. " 

implementation .. * 

effectiveness . : 



" - y • i 



Figure 4,2 Illustrative criteria for judging decision 
/Hit ernatxves, , 
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' Determining Value /Positions from Others ■ - ' 

- r Deoisien criteria are not •* right" and "wrong" as such* ~They" " 
: * represent = individuaiS- , values abotit what are. the important \ ^ 

dimensions against which to judge potential = 

.v ; ; ■ • •iri the search for jointly adde^teble (drfteria) # : : ^v;-*^ : ; / , 

^ conflict arid cooperation are intimately linked together . / ■. VH" i; ^ 

^ * The existence of con by giving ■ *p 
rise to the need for it. (Radford F 1977, pp. 74^5>) 

Decision making is made simpler wheh a consensus can be J: 
reached about criteria, but a consenstfs. is not always possible, V ^ 
nor desirable, when true value; differences exist. The DAP ; 
attest to reach agreement about the value dimensions and their 
relative importance , but they do not require agreement . . y 
Consensus may often be more illusory than real when it is \ ; 
obtained in a traditional group meeting format; Members may - ' 

engage in "group think" in which the desire to achieve consensus 
overrides the members 1 willingness or motivation to pursue ; ; - 

conflicting positions. In face-to-face meetings, not all * 
viewpoints might be expressed , because -of the r e luc tance of some • • --. 
members to speak put -in the presence of superiors, because of 
hesitancy to put forth thoughts that are feared to be unpopular 
or stupid, because o£ domination by the leader or by other 
k participants, or because hot all constituencies are represented - 
at the meetings, - 

The Delphi technique , or any of several variations of -it/ has 
been suggested in part as a way Jof reducing the distortion of 
views that face-to-face group meetings can foster, Although 
originally developed as a method for forecasting future events, */ y- 
often in situations Where little data were present, more recent 
applications generally share only these two features s > 

1. Several respondents present their views independently 
' and anonymously . ... .. 

V'-Y . "2, : Feedback is provided to the respondents , who are k : - r ' 
asked to reevaluate their original answers. 
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j For purposes of determining aritecia to judge decision 

alternatives, the following modified Delphi . technique rt, 1 1 / 

• suggested* * First, using approaches sugges ted in Figure v4.1, the 
••: - leader^ formulates and^ distributes a -list- of -possible criteria. - / : 

' The criteria should be clearly worked. Definitions and/or 
- examples-might be added to increase accurate communication of : V • 
h - meaning* Second/ respondents are asked, independently arid : ■ 
anonymously, to suggest other criteria, to rate the, criteria on • 
importance without regard to how; difficult the criteria mi)jht be ^ 
to measure ^ and to give reasons for their ratings, Third, the: ^ ;^ ,7 
... leader summarizes the responses and circulates a revised list, of ; _ 
v criteria for further ratings • . i ' -V.' '• 

* A compromise between this modified Delphi procedure -and the „ 
• usual^grou^ . is : to have the _d iscussion: take: ; place, face- „ J: 

to-f ace > but still retain the anonymous voting. A talk- judge 
pattern is followed. Leaders should encourage all to speak ("I'd 
like to hear from each of you"), should remain impartial, and 
should prompt participants to be devil's advocates, so that many 
views and suggestions have a hearing . ~ ~ :v : .-• - r 

The goal of the Delphi and similar techniques- fe* not ^ 
necessarily consensus, although relative agreement can be a 
facilitating consequence . Rather, the procedures alert 
. participants to a variety of viewpoints and challenge the , * 

participants 1 values and the assumptions on which they rest*, 
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? ^ * ^ V CiihPTEiC 5 r APPLYING DECISION MdS " 

They Jcnow the cost d£ Everything and the .value of 
: I- • -'-.^ nothing • ^ v./ ;->y. ; \ : ' "v-.jA 

- . _ . Ansel Adams # nature photographer t 

\ . .'. referring to President Reagan and 

^ Secretary of t the Interior James Watt 

: in vthis chapter, ^lt is assumed that those participating in 
?. making the decision have identified two or more potential -\" 
decision actions, and have described two or more important 
criteria (tiimehsions bf value) * (If only one criterion is ; " 

. evoked , the : dec isipQ^prpcess is tr ivi a 1 . . The decision-maker ; .;_ 

: merely selects the decision alternative that scores best on the 
single criterion of relevance* ) The DAP include aids for 
evaluating each option, against eac The procedures to 

be > descr ibed resemble most 'closely those suggested by Gardiner 
and Edwards (1975)? Edwards, Guttentag, and Snapper (1975) y 
Edwards (1977)? and Einhorn and McCoach (1978) However/ a great 
number of other procedures and variations have been proposed in 
the technical literature* Some" bf these techniques are mentioned 
and referenced >Lh Appendix A, ' 

The decision aids included in the DAP are methods that have 
met the following five criteria: ' > 

IV,. Are relatively simple to use* 

2. Yield good approximations to results obtained from 
' '•• more exacting methods* 

- 3* Are appropriate for most educational decisions, 

4* Lead to the same options being evaluated, best, 

regardless of: what other lesser alternatives are 

..considered.,- ; ". =.-; : "\. ■ 

5. Provide insight about the decision situation. 

Overview of the Decision Aids 1 * 

The steps in evaluating a decision alternative are implied by 
the form shown in Figure 5*1. A separate form is used for each 
option being considered* Once an alternative has been identified 



atr the top of _the form and the ^criteria* described in^ tiie lef tr 
most column, the nekt s'tep -is to assign* weights to the criteria, 
? by entering numbers, in Column A of Figure 5.1. ^Failure to do so 
wdiild amount to : g ivincf each dimension; equal importance , The same 
criteria ;ind cpMmri-. of weights are use^ on each form; # • 



Alter natts*ej : 



8. 
9. 

id. 
11. 

12* 



- r , Estimated Estimated Weighted 

Criteria C weight Outcome Value . Value 



• ... * " 



Value Indicator 



FigWi S.i A form* fo^ valuing a decision alternative. 



' DAP re^uiie that those- participating in; the dec ision making ; .; 
are able to project the, -future »nse^ances of Alternative . 
decision prior; to ^their selection and "implementation . This 
requirement is implicit In even casual, nonanalytio dedision-' 
making procedures • ; ^n : DAP^the ^rbjeotea consequences r j with 
Aspect to r eaoh of the criteria*., are^ placed in Coluflin- B., - Eor 
example , if -achievement loss is one .criterion, and if. the 
alternative under consideration iS: estiinated : to- result in a loss 
of six standard score units , then the number six would vbe entered 
in the appropriate row of Column B* 1^ v ^ ^ ^ ;"' •"/;-■'.- y . 

The numbers in Column B will be in different units- Some may 
be in standard score'- units , others in dollars, still others in 
rating scale points, etc. For reasons described later in this 
chapter , the validity of the DAP requires that the estimated 
outcome entries be converted to a common -yardstick of value. The 
results of this conversion are placed in Column C. 

Finally, the total estimated value for the alternative needs 
to be calculated and compared with the cor r espond ing value ; 
indicators for the other potential .decision actions being ... 
considered. The option with the greatest value indicator is' the 
one recommended for implementation. . • 

Applying decision aids involves making numerical estimates 
and manipulating these numbers* ^ The mathematical results may . 
appear to be exact arid 1 correct. The DAP user should not 
attribute more precision to the numerical values than they 
deserve, but neither should the va lues, be treated as worthless 
and ignored* Political polls are frequently off by many 
percents, but the numbers are valuable , nonetheless* The 
validity of the DAP rests in part on the belief that it is better 
to operate on the best information and judgments available , even 
though they may be in error, than to take the position that, 
because uncertainty and inaccuracies are likely, numerical , 
estimates should be disregarded. 

The steps of applying decision aids are listed as the main 
categories of Figure 5.2. Specific methods for their execution 
are provided later in this chapter . 1 "\ . ■ ; f 



I Iw-'-C:;/ Allocate jmpo'c tance weights to the or iter la ; ^ ;. : . 

% a) Assign : a ' weight of 100 to r the most; important ' ^ ^ " ^ 
:.;v='. : v"^ \ dr iter ioiv . ^ ' : ^ : : 7/;:^' ; ;,.^ ; ;;. . ,i t v-.:"? : ^V*K"-v.- 5- ^ : '-- : ?-]^- : '--. 
V.- ^^/b) ^ Give -iesser weights - to^^ther- value, dimensions # . ; ^ L 
S". depending oh their judged worth relative to .the most 
important criterion. " . ~y 
, c) Compare each secondary criterion with the other 
■ j - ; secondary criteria and adjust initial weights as.;\... 
■V^ [ • : ^' :: ^ v ••'.wa'rrmlted,^^• , '/ ... V-;"" ; v '- .>: r /: , 

d) "Allot weights to components of more global criteria* 
" : V, ^) Delete criteria with very low weights. ; ; .' 

: : f) * Perform separate analyses if near consensus on 

. weights is not reached. * '\ 

2. For each decision alte r nat i ve # * estimate outcome with •; 
respect to each criterion. . . " ; 

..A- JL : •..*• - ): : ^Qj^g it : A^gs t • • ajyige » ^jL-il L _; ■ /;• ^ : .j,v; 

bj Gather needed data* 7 
v. c) Perform separate analyses if near consensus on 
expected outcomes is not reached* " 

3* Convert outcome measures to a common scale of value. 

; . _ a) .Identify minimum and maximum plausible outcomes for 
each cr iter ion . : * - \ * 

^ b) Construct a value function over the range of ■ 

plausible outcomes. u V 

, • c) Obtain the value scale amounts from the value 

function. . '\ \ 

d) Perform separate analyses if near consensus on value 
> functions is not reached. 

4* Compute aji ove r a 11 va lue Indicator for each decision 
_ a Iter nat lye . ' ,. . • - . .' . . • y • . - •_• 

i a) Multiply a criterion's weight and expected value. 

o) Sum these products over all criteria to obtain: an.- 

r overall value indicator. v t 

o) Compare the overall value indicators of the .decision 

/ alternatives. ! 

d) Choose the alternative with the greatest value 

V indicator if the indicator is significantly higher 
.than those for tile competing Options. ... 



" Figure 5.2 ^ Steps in valuing decision alternatives * ^ 
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To help illustrate the methods, €h%-follow i ing example will' be" 
used. A school * district, faced with reduced /federal, state, and 
local support^forw its mmpensatory '.educational' prpgrafflf identified 
five wa^s> which are consistent with state regulations ^and local 
contracts V.: for cutting back on the program to bring it within the 
budgetary limits . - The options are : ,/ - 1 . ... _.,V r; >:.: v ;V : 3i^i-"' , -- : - 

1. ; Serve fewer students by changing the cutoff score for 
qualification into the program slightly for reading and 

~ " " moiTe . dramatically- for mathematics * . '-' :-/ --r-~:--^-^^- ----^ = ^ 

2, Serve fewer students by dropping some (higher) grade 
- levels and schools*; 4 w / 

3 • Reduce instructional time from five to three days per 

week- ' ' '■ - : • « ' "-. .- . * ' • • '-" , '•..*. : 

+ : -. 4 

in^class student to staff ratios and by establishing 
independent learning laboratories. : 

5, Reduce the number of professional staff needed by ... 
increasing class size and by greater use of teacher aides. 

/ We shall assume that each of these options, iand^per haps some 
combinations of them f have been described in detail. Those 
participating in the decision also described the following 
criteria /against which to judge the decision alternatives. 

1. Achievement loss. 

2. * Congruence with local policies* 

• 3. Ease ofc implementation/practicality. - '• • . • : - 

4, Support/acceptability of;;' ^ 

■ a) aides, 

b) parents'/community, 

c) students, . ■.■ ' , '• 
. * d) teachers/principals- ■ 

5. Side effects (good or bad) expected. 

Cost was not included as a criterion in this example, because all 
options weife designed to fit within a fixed budget. Since the 
options were equivalent with respect to cost , 9 cost could not be a 
basis to evaluate the\relative merits of the options. ""• 



Weighting Criteria 

" - itot Jail 5 the ec iter ia will ; .1^^ 
• it '"is recommended * that iffipoctanee weights be -' , 'assi9hedv_tb > :v.the : -'-' 
criteria* A procedure that- has worked well is- to assign a weight 
v ef 1GQ £o_ the mps;t; important Criterion:/ and lesser weights to the 
other va lua dimensions depend ing upon the ir judged worth* relati ve- 
to the most important criterion*. In the compensatory education . 
example # suppNOr t/acceptabillty|was viewed as most important and ; .; 
received a weight of 100; congruence with local pblric^es and ease 
of implement at ion/pr ac t i ca 1 i ty were seen to be half -as important 
as support/acceptability and each received a weight of 50 r 
achievement 1ms was considered only one-fifth as important and 
received an importance rating of 20; etc. (See Figure 1 5.3) Once 
an init i al set - of we ights has been .ass igned , • . it is ad visable_ to 
consider other pairings and, perhaps, make some warranted ; 
ad j ustments * For example, ease of implementation, with its " 
weight of 50, might be compared with achievement loss (weight = 
20) to consider whether the 50 to 20 (2.5 to 1) ratio still seems 
reasonable. : ' - • v • ■' ' - • ' " ' ~ : ~ ' ■ 

Note that all components ©fi support/acceptability were 
initially treated as a group, both to simplify the weighting task 
by reducing the. number of initial ratings and to increase , . 
accuracy by reducing the tendency to overweight a set of 
conceptually interdependent cr iter ia * Those participating in the 
decision process how need to divide the 100 points of weight for 
support/acceptability among the four components . The results of 
this allocation are also shown in Figure 5.3. 

At this point, consider eliminating one .or more criteria that 
have small weights (less than ,31 or 5% of the total of all the 
weights)* Their- contribution will be negligible and, will not 
materially affect the final judgments* In the example, side 
effefcts was eliminated from the analysis that followed, since its 
weight of 5 was only 2.2% of 225, the total of all the weights* 
It is because a large number of criteria cannot all; carry a heavy 
weight that the recommendation was made in Chapter 4 to limit the 
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Figure "5.3 A completed form for valuing a decision alternative 
4 for the compensatory education example. 
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criteria to those few considered important enough to affect the 
decision, ;As is the easa with the ^an^ii"; roughly 6-8 seem a 
good .number of value dimensions ? twici as many wOT-ld be too many* 
f Three methods for obtaining, the weights arej- by voting, 
(2), by verbal bargaining, and (3) by delegating the task to a 
Single person*/ Consistent with the procedures recommenced for 
determining the^ cr iter ia b^ soliciting advice from others (see 
Chapter 4) , it- is suggested (that : the weights be obtained by - /*// 
voting, with discission between rounds of voting', 

What if the voting procedures do not lead to general ' • \ 
agreement? In such a situation, separate analyses can be carried 
out, one for each distinct set of weights* Frequently the final 
results will not differ materially regardless of which set of 
weights is used. Even if the final results do differ # it is 
nevertheless very instructive to learn how the overall value 
indicators of the ^decision alternatives are" 1 affected by basic 
disagreements over the importance of the criteria. At best, the 
DAP can provide only\ increa^d illumination, not miracles that 
make conflicting ralue assumptions disappear. ^ ^ 

Estimating Outcomes ■ * ■ ' 

No -one knows for sure what will happen if a particular course 
of action is followed * n . . .all actions in the real world are 
shrouded in uncertainty, The question, is merely one of learning 
to live with uncertainty. " (Bross, 1953, p. 26) One way to live 
with uncertainty is to make uncertainty itself the attribute to 
be est una ted ( see Appendix A) f a second way , and the method 
chosen here, is both to treat extreme uncertainty or risk as a 
value dimension (see Figure 4.2) and to make the best estimates 
one can about the consequences of the decision alternatives, 

Unlike the steps of determining the criteria and assigning 
relative weights, where individuals 1 values are important and 
diversity of opinion is respected, the step of estimating 
outcomes is one for informed judgment. The, best advice and data 
are sought about what consequences can^ be expected if a 



^particuiai^ 

experience u te 11 us? What do experts and others predict ^the 
•^V. oVtporojesr wilir be? s ':.>'■. ■. v "7 7" • 

' ^ isn't estimating, outcomes subjective? Yes, it of ten is, but 
« beirj^ stto j eptivav does Vnot. mean the estimation must: be biased* • ^ 
^ Whe^ "opinions about the expected outcomes' do. 

: exist, .more information should be gathered (see Figure 2.1) . 
* When* large "differences" in expectations remain , separate analyses 
using different expected outcomes can be conducted to ascertain 
the effect that these! disagreements over predictions may have \ : . x 
regarding the overall value indicators of the decision - 
alternatives * . '• : ? ' ;.' . ; . \UY:- - S ?V:. \- 

; The entries in Column B of Figure 5.3 are the estimated 
outcomes for the compensatory education example - The entry for 
the achievement loss criterion of six is in normal curve 
equivalent units, a type of standard score; the entry for 
congruence with ; local policies is based on a 3-point;_ rating scale 
(3 - completely congruent with local policy, 1 = conflicts with 
local- policy, in an important way) ; the entry for ease of 
J implementation/practicality is based on a 5-point scaled 5 = easy 
to implement, completely practical) ? and the entries for • - 
support/acceptability represent % r atings on a 7-point scale (7 = 
fqlly supportive complete ly acceptable ) . 

Criteria for other examples of decisions might be attendance 
and usage (measured by /frequency counts), time (hours) space 
. (square feet) , etc. Cost, an important criterion for many ~ " 
decisions, is difficult to measure ,_ ^ par t ly because many factor s : 
contribute in complex ways to the calculation of cost * A brief 
discussion of the measurement of cost may be found in Appendix B. 

Connecting Outcomes .'to*. Common (Value) Units . 

It may seem odd that outcomes as diverse as achievement loss; 
congruence with local policies, etc* , can be placed on the' same 
scale of value, Yet, if you think about it, we compare the value 
of. diverse entities all the time- Would we rather spend our 
money seeing Aunt Mabel or buying a new set of golf clubs? We . 



can judge whether a hand-crafted pitcher has more value to us" 
than an equally prided private tennis lesson 'by noting which one: 
we chooser ••; :; • . •• • : / .•' ; -' 

Money is a numerical index that makes it 'possible to. compare 
? th# "yaluQ*; of - : a Wide; range of coriunodi t iesvand services :pn ~ theV 
same scale. But money is an imper f ec t measure of value . Most Q.£ 
us would prefer to win a sure $10,000 than to have a 50/50 chance 
of winning , either $0 or $20 # 000 # even though the expected 
winnings. p£ the latter arrangements are also $10,000, That- is ; 
because the value to us of the second $10 #^00 is less than the 
f irst $10 # 00 0 • The relation between riioney and value in this case 
is shown in Figure 5,4, where equal changes in dollars do not"; 
correspond to equal changes In values* \ • 




$0 $10,000 $20,000 ^ 



• ■ • . -i. ' . 51 Money : 

... .*T . " - ' " .-, . . i; 7 ' ' • ' . 

v^.-> /: ;v: : \ \ \ .: . .' 6 - v 

•" , ' , * * . . . ■ * • , , 

Figure 5*4 A value curve for money. 



; The ; nonlinear eolation that may exist between the jinits that 
outcomes are measured in and the value of the 6utcom@s is one 
reason why outcomes need to be converted to comm6n value units* 
The distortion caused by using , unconverted units is even worse 
when th% relation is shaped like an upside down u. For example, 
if the criterion, is length of the** class per iod , too long ' a ., time 
or too short a time is less valued than an intermediate length, ; 
WiAout Changing : time (in minutes) l; to a value measure, decision c: ^ 
options that called for * long class periods would be feated higher 
dn the criterion than options that called for intermediate length 
class; periods,; \v. i? . : -'* v- V 5 V : • 

'•• Perhaps the? most important reason for converting outcome 
units to common lue units is to protect the g integrity of the 
criterion weights. The true weight a criterion carries depends 



upon toe variability of . the outcome measures , Converting 
outcomes measures into the same value scale has the effect, of 
reducing differ ences, in variability among . the criteria and, thus, 
in turn, preserving the intended relative weightings of the value 
dimensions^ ^If 

instead of dollars, so that the cost entries were 100' times, 
larger (5^ 



100 times larger than if cost , wer^p measured- in dollars. A rating 
scale from 1 .to 10 would likely count twice as much as a scalfe on 



the same criterion that went from 1 to 5, 

The following^ simple 3-step procedure is suggested for 
converting any expected outcome units to the common .value scale , - 
Firsts identify minimum and maximum plausible outcomes for each 
criterion* For example, in considering, how the achievement of 
toe present students might change under ^various cutback programs, 
one might estimate that the worst plausible consequence <wou Id be 
a 15-point drop and the bkst plausible consequence would be no 
drop at all, (See Figur^ 5.5, ) , 

Second, draw a value- function for each option. This can be * 
.done merely by constructing a straight line between two points, 
one in which the worst plausible outcome is given a zero value, 
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Figure 5". 5 \ A value curve for achievement loss* 



and one in which the best plausible „ outcome is given a value of 
lOO* {For illustration, the two points are made extra large in 
the example of the achievement outcome shown in Figure 5.5. ) 
A straight line value function is often a good approximation, 
especially *when each additional unit on the expected outcome 
measure is considered * to cor respond to approximately the same 
increase in value* However, other /shaped curves can be drawn ^fc>y 
hand between the points if there is reason . to Believe that the. 
yalue does^npjburise essentially at a= s 

that shown previously in Figure 5*4; an* elongated S curve, of an 
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,v'r».. -« v: - ■/ • , \)]-:. ••• •; ;• •;• : •; ' ; ;„•••"•: ;;;;:.-.>: ' ; :'"r-// - * \vv 

inverted U curve might seem appropriate * Whether the curve is a 

straight line ,or is sketched, to form .some other shape, is a matter , — 

of the" decision '* lufeers 9 -'- judgmen't.:.' : r . : , ; . 

Third, read the value figure for a given outcome measure 
directly from the* curve, /For example^ an achievement loss of six 
standard score units was * expected under the first option : - : - 4 

considered (see Figurjr'5,3) , From the curve shown in^Figur© 5,§# ^ - 
We see that a 6-point loss corresponds to a score of 60 on the V: j • ^ 

value scale. Note: y_ if the valu^ function is a straight line, > 
the value indicator can be calculated, instead of read off a 
graph, using the formula; = . .*V; "• V'j"- % JW- T r; 

V * -, » - expected outcome - worst pla usib le outcome » iAn 

Value Indicator «: . ^ . ^ — — — — — ; T . A X 100 

best plausible outcome - worst plausible outcome 



For our example. 

Value Indicator - % M X 100 = ^tr x "i00i ^ 60 . 

Remember , the above- formula is -appropriate only if -the value . 
function is a straight line. >' \." . 



— tDnce— aga~±h, if those participating" in the decision making 
substantially disagree over the appropriate value function , 
separate ^analyses can be conducted that use the different 
functions, 1 ■ ■ > 



Compute Overall Value Indicators 

yhe computation of the value indicator for each decision 
alternative is a mechanical step, The value vindicator for an 

alternative is merely the sum, over all criteria, of the weighted 

■ { r* ' ■ ■■ '• ■■ •• • ••■ . " •' - • • •■ ■". • • 

estimated values, The overall value indicator in . Figure 5.3, ^ for 

a "., . *' '.■ ■ . . . -, '• • • ' • : . ' ^Sr . • • 

example, was calculated by multiplying the entries in Columns A 

and C for each row, and summing the products, r * 

Finally, the value indicators for each decision alternative 

can be compared;- Do not be seduc^ejl by the false precision of the 

numerical values. If the highest scoring -option does not rate, 

for example , 10% or so above the next contender, further . 
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discussion of the decision problem wbulcl bi in orSer # Beeaus 
the D^p are explicit* they Will' highlight whe^re the different 
-^cur ■ m 

communication among the parties and, ultimately and hopefully 
the quality of the decisions that are implemented.- 

Summary ' .-V/ ; 

This chapter contains suggestions for evaluating decision 
options With respect to several criteria • A separate form 
similar to /that shown in Figure 5.1 is completed for each 
alternative choice of action being considered by the decision 
makers. The pt steps for completing the form are outlined in 
Figure 5.2, and an example of the completed form is provided : 
-Figure.- -5 «r3 «- > — ^- *->--.— -.7 --^-^^ 
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APPENDIX A?v:: ^RNMIV^: ^, OAg 



Soph ie kept her eyes shut as the flow * ; - moved its 
way up through the tributaries of her mind r nearly* 
(drowning ' her reason v ; •', <■ ) - ;' : : > /• : Y \". " 



'^^^':te^i^^- ; ^S = ; v William Stvron ; Sophie »s Choice 

v* to dec is ion making is the subject of over 

• 300 articles year ly in journals of such fields as economic^sr 
psychology , management science, ^information science , and > ■ v- 

^operations research , K brief, but amply referenced, history of 
behavioral decision theory for the case of several ^criteria is ; 
of fered by Starr: and Zeleny (1977) - 7 v ? ; \ . ^ ;I 

v : Vari at ions can be found for implementing^ each of the steps: in 
the -DAP- For example, Eckenrode (1965) describes six ways of ^ 
assigning importance' weightings to the criteria . To convert 
expected outcomes to values ,. many authors recommend that a ; 
dedision' analyst present 'the decision maker with a series of 

-hypoteheMoal^c^ 
outcomes , The preferences expressed permit the analyst to 
conslruct ooriversioti curves^ simi lar to-Figure 5*4* See Keeney 
ahd Ra if fa" (1976, pp. 96-99) for a worked example. 

In cost effectiveness studies, "criteria that can be converted 
to dollar values are separated from other (effectiveness) 
criteria, For each decision alternative, ratios are ^ then formed 
in which costs (in dollars) are compared to effectiveness (in 
common value units). The DAP can be followed to obtain an 
(effectiveness') value indicator for each "option, and that 
indicator can be used in the cost/effectiveness ratio, 

The PAP require that, the expected outcomes be converted to a 
common value scale. Methods for picking the "best Vdecision 
alternatives that do not require this conversion ares 

m^. 1. Look to see if one option is better than all the others 



X 



- vdn^^l criteria, .If so, pick that option. m If not, use another 
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2." Choose the option that is clear -ly^'i^Mt ' on'- jbhe"'nbst''-- : : 
_,, important criterion. } Ties are broken by considering the second 
most ^ r.y a,^.-^. v " =■ ;^; : ^;rJ- : =^ 

3* Establish: a minimum .outcome on ^ach of iter ion that is, 
nevertheless, still satisfactory* Pick any decision alternative 
s Whose* eu toomes expected to exceed all the ?rainimui^ • . 

' ; 4fc Assign, a score to each possible outcome on each 
, criterion. : The sum, of the scores over criteria is the value 
indicator for a decision option- - This procedure, good for > ' 

^ repetitive decisions , is followed by many bank officers in ^ " 
deciding whether to approve loans. For example, two. points, if 
married , one point if lived in the same residence y fpr over three I 
years/ five points for income in the $30, 000-$40, 000 range, etc. 

is made between decision making under certainty,, decision; making, 
under risk, and decision making under conflict. The DAP, and the 
methods above, fall into the first category, because one acts as 
^ though one" is certain about the outcomes or consequences of 



taking the potential actions under .consideration.^ Vv - 7 

In decision making under X is k, the decision problem is 
conceptualized as branches from a tree or tributaries from a r 
river , in "which the initial branches or tributaries represent 
choices and the paths emanating from each choice carry the 
probabilities that the several possible consequences of following 
the course of action will occur. Thus, decision making under 



5isk, requires 7 not only that outcomes and values be estimated,, but 
that, the probabilities of the outcomes be estimated, as well. See 
Keeney and Half fa (1976) 'for a classical textbook on this 
approach. " . ' - 

In .decision making under conflict, it is assumed that the 
outcomes 'are , in part, under the control of rational opponents, 
The theory of games addresses itself to decision making under - 
conflict. "Games " , however, in this context does not suggest 
^playfulness, as many serious decisions/in times of war, in 
business competitions, etc*, are of this type, _ 
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The decision problem gan.be conceptualized as other than 
picking, the best single option out of several, explicitly 1 stated 
options. Instead, the decision problem might be more a design 
problem; namely, to cons trust a single option that contains the 
best combination of design elements , Irr the, compensatory 
education example,' the task might be to design the optimum mix of 
class size , use, of materials, deployment of teachers and aides ,;. 
inclusion of grades and subjects, etc . , to provide the roost value 
€ or the money ava ilable . Such a* problem is a classical linear 
programming problem- ' T ; 

■;■ ; Another conceptualization of the decision problem is that the 
problem is to pick not one but a combination of available options 
that together maximizes .value* . An educational example is th^ v « 
dec i s ion o f Wh i ch s ctiodls to r k eep open - and v whi ch ~ to close One 
set of procedures for handling such a problem is provided by 
Wholeben (1982) . .'- 
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•> : -{;."V ; ;= ^ (fefl^K^NT- OF COSTS /^■■■•".V.V \ z 

The following s^tementsj represent important principles 
tftfe def inition and maasurement of costs. 1 Excellent "intcoduc^or^ 
treatments, of the topic ace pr^s^ivEted by Haller (1974) and by;; 
Levin and Seidman (1981) . "-. V 

1, Costs result from decisions* In one sense, textbooks, 
teachers, and facilities do not have a cost; rather it is the, _ - 
decision about these things that has a cost: w .\ to adopt one/ , 
textbook over another, to employ a particular staffing pattern, 
and to build a new school rather than renovate an old one i 

2. Costs, are 'anything and every thing" you need to implement k - 
bhe 7 Cjdurse ot 

3. Costs include expenses for personnel, facilities, / 4 
equipment, materials, and services. A more detailed lis4. of 
cost-generating items is given in Table B\l* 

4, Most economists consider as costs the time of sty 



i the time of students , 
its argue that if one ^ 



teach era, parents, and otters* Economist 
[ option" requires more w time of people than another, the extra time 
- could have been spent in beneficial activities, and teat these 
"foregone benefits" * are a s cost to the less time efficient 
Option* Because' of the difficulty of converting to doll.ar _ 
amounts the foregone benefits resulting from loss of tine, DAP 

- treat time as a separate criterion that is valued in the same way 

as other -criteria, ^ 

5, What counts as a cost to a program depends upon whose 
viewpoint is taken. For example , the cost of the federal subsidy 
of a compensatory program is considered a cost from thef federal; 
. government's point of view but not from the school 
# ; administrator's point of view. 
....... 6, In judging, alternative options, it is necessary /to 

consider only the cdsts of elements that differ among options. 
/ If all options, for example r require the same space, the cost of 
such space does not serve to differentiate among the options and 
can be ignored. 
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Table B.l>: 



Re sou roes or Cost^enerating, Aetivi^ 

Included iri, Project Cost :V- 



Acquisition Cost 



Oper atiortal Cost 



Design of project 

Development of materials 

Evaluation design v 

P to j ect implementation 

Equipment purchase 

Project-related * • •. 

Student-related 

Mat er i a IsV and supplies 
Proj ect- related 
Student -related 

Pre- service training 

Facilities (space) 

Installation of equipment 



Proj ec t d ir ec t ion 
Evaluation 
Management support 



Salaries 
ifeachers 

Paraprof essionals 
Specialists 
Other , ; 

_- > : - ' • • • 

In-^fv^se^^ain^T^ 



/Materials and supplies 
Project-related 
St udentr related 

Equipment « 

Replacement ^ ^. 
Maintenance 

Facilities operation 
& maintenance - 

Contracted services 

Media services 

.. ; J> ■ ... ■ .-. - 

Transportation 



Source i Haggartr 197S* 
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7, Costs lie in the future If a program requites certain 
; equipment, or facilities yp\S: } already Have> the "sunken^ cost: for 

-acquiring , the equipment or facilities is not counted in the cost 
v=of\-th©~ progranu fT ; /• ''^ : :-'--\ : - : : ^'\ : {, ^ '"'/^ - - - " ' 
f ?y / "V- : 8* QcmpetinGi decision actions m^ be e^ected^;^ 

different pattern .of, costs over time. Some may h^ve lower ••' 

>initial "costs ; others lower, cost in subsequent years * A proper . 
^^omparlson of the costs of competing options, therefore, requires 
;ih aggregation of the yearly costs to § common^ point in time * 
f Computing the present value of the cost of * ¥ )the decision 

alternatives is recommended* 4 . 

9* The time frame for judging costs is important* The 

interest rates assumed in the calculations are important. The 
: relative attractiveness of alternatives depends on the "assumed - ; 

program length and interest rates - . Several calculations*, each 

based on a different set of assumptions, are advised - 

10. Several techniques are available for pricing a good or 

service. Five of these techniques are shown in Table B*2* 
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Valuation Techniquis 



Technique 



Definition 



Conditions for? -Use 



Market ^ 
^ Prices 



:- The price for van ing red lent 
in the open market- 



For market goods 



Shadow 
Price 



The value of the sacrificed 
alternatives as "indicated oys";* - 

a) tile price of an approxi- 
mate market*? or other equiva- 

:- v . — lent for a non-mar ket good 

b) people *s willingness to pay ; 
i "for an ingredient 



J For rion*-roarket 
goods - 



Joint 

Costs 



The proportion *c*f an ingredi- 
■ ent's value allocate^ 'to the -----x 
alternative under consideration 



For ingredients 
that are put to 
multiple uses 



Annualized An equal payment made annually 
Costs that accounts for multi-year 

• /projects and foregone interest 



For ingredients 
with life spans . 
of more than one 

year .' ■ - ' 



Present ~-^A single figure for a stream. 6 £. 
■ Value //^ future costs discounted at the 

v appropriate interest rate v 



For ing red ten t s 
that have ongoing 
costs (• . 



.Sources Levin and Seidman^ 1981. 
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